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ABSTRACT 

The Delphi technique was used as one instrument in an 
evaluation of the Ruppert Education Center^ an alternative high 
school* One of its contributions stemmed from its conventional use as 
a vay to a^ssess goal priorities of school polic^^makers* In addition 
it helped to (1) focus the evaluation; (2) assess community interest, 
knowledge and participation; (3) measure the congruence between the 
priorities of the faculty and ^^olicy-makers; and (4) clarify and 
operationalize goals* Osed resourcefully, the Delphi technique is a 
productive evaluation instrument* (Author) 
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Inevitably^ as projects with limited resources grow^ their ambition exceeds 
their current capability. Thusr every once in a while it becomes necessary to 
examine critically their goals and needs, to decide which areas deserve attention 
and which can be neglected. Since people within the project will advocate differ- 
ent programs, it is important for them to compromise and agree on a common and 
smaller set of priorities. 

There are different ways to have people agree on what the goals of a project 
should be. That is, there is no consensus on how to reach consensus. But while 
nethodologies differ, they share the common purpose of helping members of an 
organization agree on priorities and needs, for example. In our proposal to the 
COooiittee for a Comprehensive Education Center (CCEC) and to the Ford Foundation, 
Educational Research Corporation (ERC) suggested that the Delphi technique would 
be an effective and equitable method to help the community of the Ruppert Education 
Center (REC) achieve consensus, or at least some agreement, on its goals, for the 
alternative school it operated. Park East High School, which serves East Harlem 
and Yorkville residents. 

The Delphi technique was developed to allow each member of a group faced witli 
naking a valuative judgment to profit from the judgments of others as he reviews 
his own judgments. By cycling through the feedback operation several times, even 
groups that had disparate judgments initially tend to mcve toward consensus. This 
approach has been found superior to developing judgments in a committee session 
because both the individual judgments cind the feedback, which includes all other 
judgments not just those of the most vocal members of a group session, are anonymous. 
Thus# ERC viewed the Delphi technique as an appropriate way of obtaining the goal 
priorities of various members of the REC community. 

He also anticipated that it would aid the evaluation in at least two other ways. 
First, it would help focus the evaluation, since given the vast array of things one 
might evaluate, it would seem to make sense to concentrate on those which are seen 
to be most important to the project. Second, it would provide an unobtrusive measure 
of community involvement in, and commitment to, the project. Because we planned to 
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mail the De].phi technique forms ^ we were interested not only in the nature of the 
responses, but also the number of responses. A poor response rate would provide one 
piece of evidence that community involvement needed improvement. In short, we 
es^ected the Delphi technique not only to help the REC community reach consensus on 
its priorities but aloo help to provide direction and data to the evaluation effort. 



PROCEDURES 

REC Commxmity 

Selecting the Sample , In cooperation with Mr. Gilbert Belaval, director of CCEC, 
ERG decided that, since the project was governed by several decision-making com- 
mittees (Operating Cdmmittee, Joint Planning Committee, Representative Council r 
OC£C Executive Committee), we would poll these groups as to the relative importance 
of the vcurious subordinate goalc the school designers included in the project's 
proposal documents. We added a few other names of significcuit persons in the 
school's life or history, who were not currently on one of the aforementioned 
committees. All together the list included 57 people, whose names cuid addresses 
we obtstined from the CCEC Executive Offices. 

Most committees include Park East students. However, because the new elections 
for student representatives, which were scheduled for the week after the first 
administration of the Delp) were postponed for several weeks, no student repre- 
sentatives took paurt in the Delphi technique. 

Round One : After compiling the list of recipients, in late September, 1973, ERC 
mailed out the first round of the Delphi technique material, which included a 
cover letter, a questionnaire (response form), and an addressed, stamped envelope 
(see appendix). The cover letter indicated the role ERC was playing at REC. Because 
many people find it threatening, we were careful not to use the word "evaluation". 
Instead, the letter opened with, "This year the. Educational Research Corporation has 
been retained by the Ruppert Education Center to help identify its strengths and 
weaknesses and recommend the appropriate changes." The letter went on to state 
that there would be additional steps to tliis process and asked the recipients to 
return the forms soon after receiving them. 

Because the documents we had read implied four sets of goals, we likewise 
divided the questionnaire into four parts each corresponding to one set of goals: 
(1) School-related curriculum? (2) School-related non-curriculum; (3) Community 
Services; and (4) Community Planning. For each set, we told the participants to 
express the importance of each goal into one of four categories: (1) "most 
important," (2) "moderately important," (3) "somewhat important," and (4) "not 
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iaroortant— should not be a goal*" To force participants to set priorities and 
not respond that each goal was most important^ we added the constraint that no 
more than one-third of the goals could be placed in any one level of importance • 
We also told them to list any additional goals that we might have omitted* 

We had hoped that the forms would be returned in one week. However ^ we 
received very few and therefore called each participant to encourage them to return 
the form. While doing this^ we found that very few persons lived at their addresses 
and that in general^ the list of names ^ addresses and telephone numbers were in- 
accurate and/or out of date. Several committee members had changed jobs or moved 
residence months before and the CCEC office did not know where to send correspon- 
ence. In all^ we waited three to four weeks for the returns to Round One^ and 
still received only twenty two of the fifty seven questionnaires that we had dis- 
tributed* 

Round Two: In late October^ after we vabulated the results of Round One and 
revised the form slightly ^ we mailed it to the participants* We made three revisions. 
Firsts we modified the cover lett. r to cleirify the fact that we derived the goals 
almost verbatim from their docximentSr and that we were asking them^ as members of 
R£C*s policy making committees^ to express their priority for each goal* Second^ 
some respondents had had trouble conforming to our constraint of placing no more 
than one-third of the goals in any one of the four categories of importance* There- 
fore, to give them more flexibility in stratifying their priorities, we added a 
fifth level of importance, "very important," and inserted it between "most impor- 
tant" and "moderately important". It should be noted, that the names of the cate- 
gories are not really critical, since we were interested in the relative rankings 
o£ each goal. 

Third, and probably most important, the form for Round Two indicated the results 
of Round One* That is, for each goal we obtained a composite rating by scaling the 
levels of importance and computing the average (mean) score* The form revealed this 
conqposite rating by thrice underlining its level of importance. This procedure of 
informing each peurticipant of the judgments of other participants is basic to the 
Delphi technique. This time, then, before expressing his priority, each member knew 
how others responded and then could decide whether to conform or not. 

The response to this second round was similar to that of the first round. Again, 
ERC phoned the non-respondents to encourage them to return the forms* Despite our 
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efforts,, only twenty-four recipients, sixteen of whom had responded in Round One, 
returned their form. The responses were very consistent with those of Round One 
and, as we had initially expected, obviated a third round similar to the first two. 
Round Three ; Round Three represented cur innovation to the Delphi technique. 
Having obtained in the first two rounds the expressed priorities for each goal, we 
thought that it would be useful, for a number of reasons, to ask the participants 
what components, if any, existed to achieve each goal and whether or not any com- 
ponents should be added. 

One reason for asking the pcirticipants to do this was to get one measure of 
their knowledge of REC. In order to complete the form, they would have had to 
be familiar with the various procedures and components at REC. Since the partic- 
ipants were on policy making committees, one would hope they would be knowledgeable 
about the program for which they were making decisions. 

Moreover, because of the vagueness or generality of some of the goals in the 
docximents, we thought that it would be useful to have the participants tackle the 
prpblem^ of identifying the operational dimensions of each goal. That is, while 
some of the goals were couched in rhetorically appealing terms— -e.g. ••to make racial 
integration an educational asset** and ••to restore to teach' rs some sense of 
virility*' — they appeared to be so vague or all-inclusive that it would be difficult 
even for someone familiar with the program at REC to pinpoint specific components 
to achieve them. Thus, we saw this exercise as one way of getting the participants 
to confront the issue of translating goals into policy and rhetoric into action. 

A third reason for this exercise, was to provide some data that would help 
determine if there was consistency between the importance of the goals and the 
measures being employed to achieve them. Because any program has finite resources 
of personnel, time, and money, it presumaibly should be investing the most resources 
in those goals which it considers most important. Thus we wanted to determine, 
from the perspective of policy makers, if REC has identifiable components to achieve 
its more important goals. In effect, we were asking the participants the question: 
To what extent do the resources match the priorities? 

Our fourth reason was somewhat selfish. We thought that by having the partic- 
ipants identify components it would help us focus our evaluation efforts. For 
example, if they, as in fact they did, considered the most important school-related 
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curriculxun goals to be each student's learning to read and understand mathematical 
concepts, and listed the basic skills courses as a component to achieve this goal<> 
then it would make sense for us, as we did, to analyze the math and reading test 
scores. VJhenever possible, we wanted to have correspondence between the eval- 
uation effort and the priorities of REC. 

In short, we had ambitious reasons for Round Three. We realized it was a 
relatively formidable task to complete the form. We decided, therefore, to mail 
out the form to only those thirty participants who had expressed some interest 
by responding to at least one of the first two rounds. Even so, we anticipated 
a low response rate, and we were correct. Only six people returned them, and even 
some of these were incomplete. Most participants, then, did not, and perhaps, could 
not or would not, complete the forms. 



Park East High School Faculty 

For most of the reasons (e.g. to help us identify components, to translate 
vague goals) given in the previous section, we also used the school-related curric- 
ulum and school-related non-curriculxam sections of our Delphi technique forms with 
the faculty at Park East. An additional reason, however, was to determine the 
congruence between the priorities of the members of policy-making committees (some 
of which do include a few faculty) and the faculty at Park East. One would hope that 
those who are responsible for implementing many of the policies have priorities 
similar to those who are responsible for making the policies. 

While the forms we used for both groups were the same, the conditions under 
which we administered them were not. During mid-year at a staff training session, 
we administered the goal-'component form (Round Three) to the faculty. To avoid 
duplication of effort and ease the task, we had them work in small groups. We had 
two other considerations for administering the form under these conditions. First, 
it seemed to be an appropriate use of a staff training session. Second, by having 
them do it there, we would be assured of a high response rate. 

Later in the year, as part of a teacher questionnaire, we administered the 
form for the first round of the Delphi technique at a staff -training session. We 
asked for the same information from, and imposed the same constraints on, the faculty 
as we had from the committee members. 

gp^(^" ^ Educational Research Corporation 



RESULTS 



As mentioned previously^ early in the Delphi technique procedure ERC found 
various threads of evidence that suggested that actual community participation 
and communication did not match that outlined in the vcurious documents issued by 
CCEC* First, there was a low response rate to the Delphi forma. Approximately 
40% of the people who received forms for the first two rounds responded, despite 
our (1) enclosing a stamped, addressed envelope, (2) offering to answer questions 
and providing a telephone niimber to do so, and (3) following-up the non-respondents 
with phone calls* In Round Three, the response rate was only 20%, even though forms 
were sent only to those who had responded to at least one of the first two rounds. 
This low response is partly explained by the fact that the task was quite difficult 
and required some time. Tciken as a whole, though, the response rate to the Delphi 
technique reveals some problems with the committees, whose members presumably have 
a greater commitment to REC than does the community at large. 

Part of the low response rate probably can be attributed to a lack of commun- 
ication between CCEC and the members of the committees ♦ We have already mentioned 
that many of the addresses and telephones were out-dated. Worse, many of the names 
we had received as being members of committees were no longer members. Clearly, 
the records were in poor condition — in some cases incomplete; in others, inaccurate. 
We also found, much to our surprise, that some members were unaware of the evalu- 
ation effort. In short, then, a combination of factors, including the communication 
between CCEC and tha members of committees, outdated membership lists, and lack of 
interest, probably contributed to the low response rate to the Delphi technique. 

Among those committee members who did respond, there was general agreement on 
the priorities of the project. For instance, almost everyone agreed that the teach- 
ing of basic skills in reading and writing were the most important school-related 
curriculum goals. Further, most people felt that the community planning goals that 
provide for community participation and membership on committees and inform them of 
policy decisions were more important than the other goals. 
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Perhaps equally important, there also was general agreement on the priorities 
between the committee members and the faculty at Park East. As was mentioned 
earlier, one always hopes that the views of policy makers are congruent to those 
who implement the policy. Otherwise a range of tinwanted problems can emerge. 
Summary tables showed that committee members and teachers had similar priorities on 
the two sets of goals to which both responded. Rank correlation coefficients were 
statistically significant, reflecting the convergent opinions of the two groups* 

This agreement is a positive finding although one must remember that it does 
not necessarily follow that because people agree on goals, they also agree either 
on how they are being achieved or on how to achieve them. ?ound Three, the goal- 
coit^nent linkage, clearly reflected this. In most instances, the components were 
listed by more than one respondent. Many other responses were idiosyncratic. In 
other words, the respondents often did not agree on the components that existed to 
achieve the goals, whether or not a component existed, whether or not components 
should be added, and, if so, what these should be. 

There are at least three factors which contribute to the variance in responses. 
First, the respondents differ in their familiarity with REC. Thus, some may be 
unaware of certain programs and therefore claim there is no component, when there is 
one. On the other hand, some may be very familiar with REC and know that a program 
that superficially appears to contribute to a goal, in fact, does not. Second, 
people have a loose definition and will name a component even if it is only tangen- 
tially relaxed to the goal, others require a clearer relationship between the two. 

Major difficulties arose with those goals that were vague and lacked expectations 
that were specific enough to be identified with programs. Some respondents commented 
that these goals were idealistic or that the wording was rhetoric. Consequently, 
the respondents found the task of matching components difficult, and some resorted 
to naming components, that while related, seemed at best to achieve only a narrow 
interpretation of the goal. Vagueness led to some providing their own operational 
definitions of the goal and thus to their naming different components. This experience 
demonstrated to those participating the necessity for clarity in prescribing goals. 
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CONCLUSIONS 

ERC had an ambitious rationale for the Delphi technique for use in evaluation 
and added the modified Third Round, the goal-component linkage, to provide additional 
feedback to the client. There were finally five ways in which the Delphi technique 
proved useful in the evaluation project. 

First, the conventional use of Delphi did provide an affirmation and a current 
assessment of the priorities of the school. We were able to rank the various goals 
in a priority order. 

Second, knowledge of the priorities enabled us to focus the evaluation effort. 
For example, since the knowledge of basic skills was clearly a high priority, we 
conducted an analysis of test scores in reading and mathematics. 

The Delphi process also provided a measure of the community involvement in the 
school when we discovered some representatives were no longer at their supposed 
addresses. Our addition of the goal-component linkages to the process also enabled 
us to assess the knowledge of the school among those who did respond. 

Having to match goals with program components helped to convince community 
representatives of tha need to operational ize some of the goals and to avoid simple 
rhetoric* 

Finally, use of the Delphi technique with faculty provided the means for 
assessing the congruence between the priorities of the faculty and of the policy 
makers. A lack of congruence would have provided eeirly an indication of problems, 
but fortunately there were no such problems in this instance. 

For all these reasons we have found that, used resourcefully, the Delphi 
technique has been established as a useful tool in the evaluation of alternative 
schools. 
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